The present invention is related to methods and apparatus for capturing and displaying images and, more particularly, to methods and apparatus for capturing and displaying stereoscopic representations of panoramic images.
Ordinary rectilinear photographs provide faithful reproductions of real-life three-dimensional (3D) scenes. The resulting rectilinear images, however, appear very flat because much of the depth information, that is, the distances from the camera to the elements of the scene, is lost in the process of capturing and printing these images. Consequently, one simply xe2x80x9clooks atxe2x80x9d ordinary photographs and there is little sense of being immersed in the scene represented in the photographic image.
Two well-known techniques can be used to enhance the sense of immersion in a photographic or computer generated scene. One of these techniques is stereoscopic imaging in which differing rectilinear images are presented to a viewer""s left and right eyes. The differences between these images can be detected by the human visual senses to provide a perception of the relative distances to various elements within a scene. The other technique is panoramic imaging in which a complete 360 degree representation of a scene is captured. Computer software can be used to project an interactively selected portion of such a panoramic image into a rectilinear image, thereby enabling a person to xe2x80x9clook aroundxe2x80x9d the scene. The ability to select any view direction interactively over a full 360 degree range provides an alternative sense of immersion. Since stereoscopic images and panoramic images both provide means of allowing a person to experience a sense of immersion in a scene, it would be useful to enhance this sensation further by combining both techniques to allow a person to see stereoscopic views of a panoramic scene.
Numerous devices have been developed for stereoscopic viewing of rectilinear images, including, for example, U.S. Pat. No. 3,850,505 entitled xe2x80x9cUniversal Stereoscopic Viewing Device,xe2x80x9d and U.S. Pat. No. 4,221,462 entitled xe2x80x9cStereoscopic Viewer for Panoramic Camera Film,xe2x80x9d the disclosures of which are incorporated by reference herein. Typically, such devices are based on capturing two rectilinear images of a scene, one of which is presented to a viewer""s right eye, and the second of which is presented to the viewer""s left eye. It should be noted that this process is divided into two distinct steps. In the first step, the scene is captured. In the second step, the captured scene is presented to the viewer. These two steps may take place simultaneously, as with the use of video cameras and video display devices. Alternatively, a persistent medium such as video tape, photographic film, or digital data may be used to capture the scene, allowing the scene to be viewed at a later time.
The scene capture step, as represented in FIG. 1, typically involves two cameras separated by some distance (D) perpendicular to the view direction (V). The distance D is defined by the difference between the location of the nodal point 10 of the lens on the right camera (R) and the location of the nodal point 12 of the left camera (L). The point M in FIG. 1 is the midpoint of the line joining these nodal points. The scene view direction (V) is determined by a vector perpendicular to the line joining the nodal points. The camera view directions (Vleft and Vright) of the left and right cameras are defined by vectors which are perpendicular to the film plane of each camera and which pass through the nodal point of the lens of the corresponding camera. The camera view directions may be parallel to the scene view direction, as shown in FIG. 1, or they may be converged to meet and cross in the area of the scene.
In an alternative means of capturing stereoscopic images, a single camera is used. In this case, the camera is placed in the first (left or right) position and a first image is captured. The camera is then moved to the second position where it is used to capture the second image. Otherwise, this method is equivalent to the use of two cameras.
It is also possible to use rectilinear images to capture a panoramic representation of a scene. Such a technique is illustrated in FIG. 2. This image capture technique may be accomplished by using a single camera to capture a first image (I1) with view direction V1 and field of view, xcex8fov. The field of view angle is determined by the focal length of the lens and the width of the image captured by the camera. The camera may then be rotated by an angle xcex8rot about an axis passing through the nodal point of the lens to obtain a new view direction V2, and a second image (I2) may be captured with the camera in the new view direction V2.
If the rotation angle, xcex8rot, is less than the field of view angle, xcex8fov, then a portion of the left side of image I2 will overlap a portion of the right side of image I1. If image I2 results from rotating the camera about the nodal point of the lens, then the overlapping portions of images I1 and I2 will contain redundant information. In this case, these two images may be projected onto a common surface, such as a cylinder centered on the axis of rotation. The resulting pair of projected images may be merged into a seamless composite image (I12) representing a total field of view of (xcex8fov+xcex8rot). This process may be repeated until the resulting field of view encompasses 360 degrees, forming a complete and seamless panoramic image of the scene.
If the camera is rotated about any point other than the nodal point of the lens, then the overlapping portions of images I1 and I2 will not contain equivalent representations of the scene. For example, occlusion relations will change, making it possible to see elements of the scene in image I2 that are not visible in image I1. If these images are projected onto a common surface, the overlapping portions of the resulting projected images I1xe2x80x2 and I2xe2x80x2 will not match and it will not be possible to combine these images into a seamless composite. In this case, it is not possible to produce a seamless panoramic image of the scene.
Consequently, it can be seen that, in order to combine two or more rectilinear images into a seamless panoramic image, the camera must be rotated about the nodal point of the lens, and the camera must not be rotated about an axis which does not pass through the nodal point of the lens.
It is also possible to capture a seamless panoramic image by using a rotating panoramic camera such as that described in U.S. Pat. No. 4,241,985, the disclosure of which is incorporated by reference herein. As with the capture of rectilinear images, this device depends on the ability to rotate about the nodal point of a lens, and the resulting panoramic image corresponds to a projection of the scene onto the surface of a cylinder having a central axis passing through the nodal point of a lens.
Returning to the capture of stereoscopic images, it is possible to attempt to capture a stereoscopic panorama by rotating a pair of cameras about the midpoint (M) between the nodal points, as shown in FIG. 3. As illustrated, the pair of cameras, L and R, capture images at first positions A, are rotated about the midpoint M, and then capture images at second positions B. In this case, the point M does not coincide with the nodal point of either lens. Consequently, the resulting rectilinear images cannot be combined to form a seamless panoramic image. An example of this procedure has been attempted by Charles Wiltgen (http://www.quicktimefaq.org/). The results of the effort are presented at http://www.quicktimefaq.org/qtvr/archive/Goodies/3Doffice/StereoPano.html, where one can easily see that large image alignment errors are present at the seams between the constituent images.
As an alternative to rotating about the midpoint M, the vector joining the nodal points can be rotated about some other point, such as the nodal point of the left camera lens, as shown in FIG. 4, or the right camera lens. As illustrated in FIG. 4, the pair of cameras, L and R, capture images at first positions A, are rotated about the nodal point of the left camera, and then capture images at second positions B. This could allow the creation of a left-camera panorama, or a right-camera panorama, but not both, since this vector cannot rotate about both nodal points simultaneously, while keeping both nodal points fixed in space. For example, the camera rotation indicated in FIG. 4 can be used to produce a seamless panoramic image for the left camera, but the images captured by the right camera cannot be combined because this camera is not being rotated about its own nodal point.
As another alternative, the internodal vector V can be kept fixed while rotating both cameras about their respective nodal points, as illustrated in FIG. 5. As illustrated, the pair of cameras, L and R, capture images at first positions A, are each rotated about their respective nodal points, and then capture images at second positions B. In this case, the view directions do not remain perpendicular to the internodal axis. This results in a progressive loss of stereoscopic separation, defined as the projection of the internodal axis on a line perpendicular to the view direction, or Dxe2x80x2=D cos xcex8rot. After rotating both cameras by 90 degrees, the resulting view vectors will be co-axial, resulting in complete loss of stereoscopic separation (Dxe2x80x2=0). Rotation by 180 degrees would result in the left camera (L) capturing an image appropriate for the right eye, and the right camera (R) capturing the image appropriate to the left eye (Dxe2x80x2=xe2x88x92D). Consequently, it can be seen that the camera orientation requirements for capture of stereoscopic images conflict with the camera orientation requirements for panoramic image creation.
As yet another alternative means of creating a stereoscopic panorama, one could employ two rotating panoramic cameras with rotation axes separated by the same internodal distance used for conventional stereoscopic photography. The result, however, would be equivalent to the situation for rectilinear cameras presented in FIG. 5. That is, as the view angle increases from zero to 90 degrees, the stereoscopic separation drops from the initial value to zero, leading to a loss of stereopsis. Stereopsis is defined as the ability of a viewer to discern the distance to elements of a scene based on the differences between the images seen by the left and right eyes. Rotation by 180 degrees leads to negative stereoscopic separation (left/right reversal, Dxe2x80x2=xe2x88x92D), and rotation by 270 degrees returns to null stereoscopic separation.
The foregoing considerations demonstrate that the conventional means of capturing stereoscopic images conflict with the conventional means of generating panoramic images. Since stereoscopic images possess valuable properties and panoramic images possess additional valuable properties, it is desirable to obtain both sets of properties simultaneously in the form of stereoscopic panoramic images. This invention, as described below, presents a practical means of achieving this objective.
Before describing this invention, two additional possible means of creating stereoscopic panoramic images are considered. These methods are presented in a set of web pages starting with http://www.geocities.com/ResearchTriangle/Thinktank/9588/. These pages were posted by a person identified only as xe2x80x9cEekmanxe2x80x9d who claimed to have done the work under the supervision of Prof. Gale E. Spring of Melbourne, Australia. One of the two methods presented by Eekman is based on using a curved mirror to capture a complete 360 degree panorama in a single image. The mirror is then to be moved vertically before capturing a second image. This approach, by itself, is likely to be ineffective because the resulting pair of images represent a vertical displacement, while normal stereoscopic image viewing requires a horizontal displacement. This is a consequence of the horizontal separation between a viewer""s eyes. No means are disclosed for deriving appropriate horizontal displacements from the captured vertical displacements.
The second method presented by Eekman is based on a rotating digital scanner with an arrangement of mirrors to provide left and right displacements. The digital scanner is analogous to the film based device described in U.S. Pat. No. 4,241,985, the disclosure of which is incorporated by reference herein. By capturing the images in narrow vertical strips, the problems of aligning overlapping portions of adjacent rectilinear images is avoided. However, one critical weakness to this approach is the need for unique specialized hardware, instead of being able to use conventional photographic or digital cameras to capture the scene.
As shown in the background description above, there is a need for methods and apparatus for obtaining stereoscopic representations of panoramic scenes. The present invention provides such methods and apparatus by using a data structure called a xe2x80x9cdepth map,xe2x80x9d as will be explained in detail below. Thus, in accordance with the invention, a scene is represented by a panoramic color image and a panoramic depth map, instead of by two color images, one for the left eye and one for the right eye. The color image and depth map used in this invention represent the panoramic color data and corresponding depth values for the nodal point of the lens of a camera rotating about this nodal point. The panoramic color image may be used directly to determine view-dependent color images for one eye in the conventional manner. The color image for the other eye is then estimated based on the panoramic color image and the associated depth map. Alternatively, the panoramic color image may be considered as representing a point midway between two eyes and view-dependent color images may be estimated for both eyes based on the panoramic color image and depth map.
The present invention also provides various camera arrangements using one or more cameras, as will be explained, for capturing image information that may be used to generate and display stereoscopic representations of panoramic images.
These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.